Online media offer a number of advantages over traditional advertising distribution techniques; most importantly, the speed of execution and delivery, and the verifiability of content performance. However, the technical sophistication of marketing techniques has lagged behind that of the distribution and consumption mechanisms upon which they rely, and many marketers have yet to fully realise the unique benefits of the online space.
The increasing prominence of web analytics and optimisation solutions in the suite of tools offered to online marketers is a sign that techniques are becoming more sophisticated, and that the marketing community is discovering the improvements in strategy and execution attainable via constant measurement and experimentation.
The performance of online marketing campaigns (including a coordinated set of email broadcasts, website landing pages, or banner advertisements) is often tracked in terms of click-through rates (the proportion of recipients of a particular content item who click on it), and conversion rates (the proportion of recipients of a particular content item who go on to complete a target action, such as purchasing a product). In order to improve these metrics, a number of optimisation processes are used, such as A/B testing or multivariate testing. However, these optimisation methods are labour-intensive and require a considerable amount of data to be processed over the course of a marketing campaign. In addition, campaigns consisting of only a small number of touch-points or media, or those consisting of a series of very different touch-points or media, are difficult to optimize because language and images shown to perform better in previous content may not be applicable to future content, and optimization can only occur between one iteration of a campaign and a subsequent iteration with similar content and purpose.
Accordingly, several methods intended to maximise the performance of online content have been developed, and a selection of these will now be discussed.
A/B testing (otherwise known as “split testing”) involves the usage of two or more mutually exclusive items of content intended to provoke the same candidate response. The success probability of each content item is based on the ratio of target action performances to content item usages, and the “winner” is the item with the highest proportion of desired actions to recipients (i.e. the “response rate”). The winning content item is then used in future marketing propositions. Although A/B testing is straightforward to understand and apply, it suffers from a number of limitations. Chief among these is the trade-off between the number of content parameters to be tested, and the accuracy of the results. For instance, if only one content parameter is altered from content item to content item, the results clearly indicate the relative performance of each selection of that parameter. Exhaustive testing of all combinations of content parameters of a complex content item one-by-one may require considerable time, and/or a very large number of samples. Conversely, varying combinations of several content parameters from content item to content item results in uncertainty regarding the true performance impact of each content parameter and selection.
Although more complex than A/B testing, multivariate testing is becoming increasingly popular and generally takes the form of either fractional-factorial testing or full-factorial testing.
Fractional-factorial testing involves the manual or automated selection of a unique subset of combinations of content items designed to reduce the number of content combinations which need to be tested, and thus the cost of the experiment is reduced. Testing on the subset of combinations is performed as in the A/B testing methodology outlined above. However, the increased depth of analysis potentially enables greater gains in response rate to be achieved. The major limitation of this testing method is that interaction effect analysis is restricted, a priori, to a set depth by the experimental design.
Full-factorial multivariate testing involves the generation of a unique, exhaustive set of combinations of content items, and an experimental methodology essentially equivalent to the A/B testing methodology outlined above. The major drawback of the full-factorial testing method is that it requires significantly more data for its results to achieve statistical significance.
There remains a need for methods and systems providing improved testing of content performance in relation to recipient response rates.